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DESCRIPTION 

MOTION IMAGE CODING APPARATUS, AND CONTROL METHOD AND 
PROGRAM OF THE APPARATUS 

5 TECHNICAL FIELD 

The present invention relates to a motion image 
coding apparatus which codes a motion image by 
executing motion compensation for frame data to be 
coded by referring to a plurality of frame data in the 
10 motion image, and a control method and program of the 
apparatus . 

BACKGROUND ART 

Recently, the H.264 coding method is attracting 

15 attention as a new motion image coding method. This 

coding method is developed by the cooperation of ITU-T 
and ISO. This new standard was standardized in the 
summer of 2003. 

The characteristic features of this new coding 

20 method are that 4x4 integer transformation is used, 
and a plurality of intra-predictions are prepared, 
unlike in the conventional MPEG-1, MPEG- 2, and MPEG- 4 
coding methods. In addition, an intra- loop filter is 
used, and motion compensation is performed by seven 

25 types of sub-blocks. Also, the pixel accuracy of the 
motion compensation is the same as the MPEG- 4 coding 
method, i*e,, the motion compensation can be performed 
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by 1/4 pixel accuracy. Furthermore, universal 
variable -length coding or context adaptive 
variable -length coding is used as entropy coding. 

The more important characteristic feature is as 
5 follows. That is, the MPEG- 1, MPEG- 2, and MPEG- 4 

coding methods perform motion compensation by using two 
reference images (frames) before and after a frame to 
be coded. However, this new coding method can use a 
larger number of reference images. A num_ref_frames 

10 code contained in the header of a bit stream can take a 
maximum of 16 values. 

More specifically, in motion compensation, 16 
frames before and after a frame to be coded can be 
referred to as reference images . A macroblock to be 

15 coded is processed as follows. As described above, a 
prediction error is calculated by 1/4 pixel accuracy 
for seven types of sub-blocks with respect to an image 
having a maximum of 16 frames, and a macroblock by 
. which this prediction error is a minimum is selected. 

20 This largely increases the coding efficiency. 

The arrangement of the conventional motion image 
coding apparatus using the H.264 coding method will be 
explained with reference to Fig. 13. This arrangement 
is also explained in reference 1 ("Overview of the 

25 H.264/AVC Video Coding Standard" (IEEE TRANSACTIONS ON 
CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, JULY 2003)) 
or reference 2 (^ H.264 Now Starts, Outrivaling 
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Slow-Going MPEG- 4" (Nikkei Electronics 2003-7,7, pp. 65 
- 74)). 

Fig. 13 is a block diagram showing the 
arrangement of the conventional motion image coding 
5 apparatus . 

Image data is input macroblock by macroblock to 
this motion image coding apparatus. A selector 1000 
switches whether to perform intra-coding. If 
intra- coding is to be performed, the image data is 

10 input to an intra-predictor 1001. The intra-predictor 
1001 performs prediction in nine modes, and calculates 
a prediction error. 

If coding to be performed is not intra-coding, 
the image data is input to a differential unit 1002 

15 where the difference from a predicted image is 
calculated as a prediction error. 

A transformer/quantizer 1003 transforms the 
calculated prediction error into an integer of 4x4 
pixel blocks, and quantizes the obtained coefficient. 

20 This quantized efficient as the result of quantization 
undergoes variable -length coding performed by an 
entropy coder 1004, and is output to an output unit 
1014. At the same time, the quantization result is 
input to a dequantizer/invert transformer 1005 to 

25 restore the prediction error, and this prediction error 
is added to the predicted image by an adder 1006. The 
result is suitably stored as a decoded image in frame 
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memories 1007 to 1010. 

A motion estimator 1011 compares the decoded 
image stored in the frame memories 1007 to 1010 with 
the input image, and calculates a motion vector by 1/4 
5 pixel accuracy for each sub-block. These motion 

vectors and the selected frame numbers are input to a 
motion compensator 1012, and reference images are 
loaded from the corresponding frame memories. A 
reference image having a minimum prediction error is 

10 selected and output as a predicted image to the 
differential unit 1002. 

The motion vectors and selected frame numbers are 
also input to a motion coder 1013 and coded, and the 
coded data is output to the output unit 1014. The 

15 output unit 1014 shapes this coded data in accordance 
with a format, and outputs the shaped data. 

Unfortunately, a coding method which refers to a 
plurality of frames such as the H.264 coding method 
described above poses the problem that motion vectors 

20 are searched for in order to execute motion 

compensation, and the calculation amount becomes 
enormous as the number of reference images increases. 

In addition, especially when the H.264 coding 
method is used in an image sensing apparatus such as a 

25 video camera, the entire image largely changes whenever 
the video camera is panned or tilted. Therefore, 
although the frequency of reference of data of 
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temporally separated frame images decreases, this data 
must be held* 

DISCLOSURE OF INVENTION 
5 The present invention has been made in 

consideration of the above situation, and has as its 
object to provide a motion image coding apparatus 
capable of efficiently using a memory used for motion 
image coding, and capable of performing efficient 

10 motion vector search, and a control method and program 
of the apparatus. 

According to the present invention, the foregoing 
object is attained by providing a motion image coding 
apparatus which codes a motion image by executing 

15 motion compensation for frame data to be coded by 

referring to a plurality of frame data in the motion 
image, comprising: detecting means for detecting a 
motion of an imaging device; a plurality of storage 
means for storing the plurality of frame data; 

20 selecting means for selecting, from the plurality of 
storage means, on the basis of motion information 
detected by the detecting means, storage means for 
storing reference frame data to be referred to when the 
frame data to be coded is coded; estimating means for 

25 estimating a motion vector on the basis of the 
reference frame data stored in the storage means 
selected by the selecting means and the frame data to 
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be coded; coding means for coding the frame data to be 
coded by using the motion vector estimated by the 
estimating means; and output means for outputting the 
coded data which is coded by the coding means. 
5 In a preferred embodiment, the detecting means 

detects the motion of the imaging device on the basis 
of a motion image sensed by the imaging device. 

In a preferred embodiment # wherein the selecting 
means comprises control means for controlling 

10 write/read and power supply to the plurality of storage 
means on the basis of the motion information detected 
by the detecting means. 

In a preferred embodiment , further comprises 
setting means for setting an image sensing mode of the 

15 imaging device, wherein the selecting means comprises 
control means for controlling write/read and power 
supply to the plurality of storage means on the basis 
of the image sensing mode set by the setting means • 
In a preferred embodiment, the control means 

20 stops power supply to storage means not selected by the 
selecting means. 

In a preferred embodiment, the apparatus further 
comprises search range control means for controlling a 
motion vector search range of the estimating means on 

25 the basis of the motion image detected by the detecting 
means . 

According to the present invention, the foregoing 
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object is attained by providing a motion image coding 
apparatus which codes a motion image by executing 
motion compensation for frame data to be coded by 
referring to a plurality of frame data in the motion 
5 image , comprising: setting means for setting an image 
sensing mode of an imaging device; a plurality of 
storage means for storing the plurality of frame data; 
selecting means for selecting, from the plurality of 
storage means, on the basis of the image sensing mode 
10 set by the setting means, storage means for storing 
reference frame data to be referred to when the frame 
data to be coded is coded; estimating means for 
estimating a motion vector on the basis of the 
reference frame data stored in the storage means 
15 selected by the selecting means and the frame data to 
be coded; coding means for coding the frame data to be 
coded by using the motion vector estimated by the 
estimating means; and output means for outputting the 
coded data which is coded by the coding means. 
20 In a preferred embodiment, the selecting means 

comprises control means for controlling write/read and 
power supply to the plurality of storage means. 

In a preferred embodiment, the control means 
stops power supply to storage means not selected by the 
25 selecting means. 

In a preferred embodiment, the detecting means 
detects a motion of the imaging device on the basis of 



WO 2005/062624 



PCT/JP2004/018666 



a motion image sensed by the imaging device. 

According to the present invention , the foregoing 
object is attained by providing a motion image coding 
apparatus which codes a motion image by executing 
5 motion compensation for frame data to be coded by 
referring to a plurality of frame data in the motion 
image, comprising: input means for inputting control 
information which controls an imaging device; storage 
means for storing a motion image sensed by the imaging 

10 device; setting means for setting the number of 

reference frame data to be referred to when the frame 
data to be coded is coded, on the basis of motion 
information of the imaging device, which is acquired on 
the basis of the control information input by the input 

15 means; acquiring means for acquiring reference frame 
data corresponding to the number of reference frame 
data set by the setting means; estimating means for 
estimating a motion vector on the basis of the 
reference frame data acquired by the acquiring means 

20 and the frame data to be coded; coding means for coding 
the frame data to be coded by using the motion vector 
estimated by the estimating means; and output means for 
outputting the coded data which is coded by the coding 
means . 

25 In a preferred embodiment, the apparatus further 

comprises search range control means for controlling a 
motion vector search range of the estimating means on 
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the basis of the motion information . 

According to the present invention, the foregoing 
object is attained by providing a control method of a 
motion image coding apparatus which comprises a 
5 plurality of storage units for storing a plurality of 
frame data in a motion image, and codes the motion 
image by executing motion compensation for frame data 
to be coded by referring to frame data stored in the 
plurality of storage units, comprising: a detection 

10 step of detecting a motion of an imaging device; a 
selection step of selecting, from the plurality of 
storage units, on the basis of motion information 
detected in the detection step, a storage unit for 
storing reference frame data to be referred to when the 

15 frame data to be coded is coded; an estimation step of 
estimating a motion vector on the basis of the 
reference frame data stored in the storage unit 
selected in the selection step and the frame data to be 
coded; a coding step of coding the frame data to be 

20 coded by using the motion vector estimated in the 

estimation step; and an output step of outputting the 
coded data which is coded in the coding step. 

According to the present invention, the foregoing 
object is attained by providing a control method of a 

25 motion image coding apparatus which comprises a 

plurality of storage units for storing a plurality of 
frame data in a motion image, and codes a motion image 



WO 2005/062624 



- 10 - 



PCT/JP2004/018666 



by executing motion compensation for frame data to be 
coded by referring to frame data stored in the 
plurality of storage units, comprising: a setting step 
of setting an image sensing mode of an imaging device; 
5 a selection step of selecting, from the plurality of 
storage units, on the basis of the image sensing mode 
set in the setting step, a storage unit for storing 
reference frame data to be referred to when the frame 
data to be coded is coded; an estimation step of 

10 estimating a motion vector on the basis of the 
reference frame data stored in the storage unit 
selected in the selection step and the frame data to be 
coded; a coding step of coding the frame data to be 
coded by using the motion vector estimated in the 

15 estimation step; and an output step of outputting the 
coded data which is coded in the coding step. 

According to the present invention, the foregoing 
object is attained by providing a control method of a 
motion image coding apparatus which comprises a storage 

20 unit for storing a motion image, and codes the motion 
image by executing motion compensation for frame data 
to be coded by referring to frame data stored in the 
storage unit, comprising: an input step of inputting 
control information which controls an imaging device; a 

25 setting step of setting the number of reference frame 
data to be referred to when the frame data to be coded 
is coded, on the basis of motion information of the 
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imaging device, which is acquired oil the basis of the 
control information input in the input step; an 
acquisition step of acquiring reference frame data 
corresponding to the number of reference frame data set 
5 in the setting step; an estimation step of estimating a 
motion vector on the basis of the reference frame data 
acquired in the acquisition step and the frame data to 
be coded r a coding step of coding the frame data to be 
coded by using the motion vector estimated in the 

10 estimation step; and an output step of outputting the 
coded data which is coded in the coding step. 

According to the present invention, the foregoing 
object is attained by providing a program for 
implementing control of a motion image coding apparatus 

15 which comprises a plurality of storage units for 

storing a plurality of frame data in a motion image, 
and codes the motion image by executing motion 
compensation for frame data to be coded by referring to 
frame data stored in the plurality of storage units, 

20 comprising program codes of: a detection step of 

detecting a motion of an imaging device; a selection 
step of selecting, from the plurality of storage units, 
on the basis of motion information detected in the 
detection step, a storage unit for storing reference 

25 frame data to be referred to when the frame data to be 
coded is coded; an estimation step of estimating a 
motion vector on the basis of the reference frame data 
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stored in the storage unit selected in the selection 
step and the frame data to be coded; a coding step of 
coding the frame data to be coded by using the motion 
vector estimated in the estimation step; and an output 
5 step of outputting the coded data which is coded in the 
coding step. 

According to the present invention, the foregoing 
object is attained by providing a program for 
implementing control of a motion image coding apparatus 

10 which comprises a plurality of storage units for 

storing a plurality of frame data in a motion image, 
and codes a motion image by executing motion 
compensation for frame data to be coded by referring to 
frame data stored in the plurality of storage units, 

15 comprising program codes of: a program code of a 

setting step of setting an image sensing mode of an 
imaging device; a program code of a selection step of 
selecting, from the plurality of storage units, on the 
basis of the image sensing mode set in the setting 

20 step, a storage unit for storing reference frame data 
to be referred to when the frame data to be coded is 
coded; an estimation step of estimating a motion vector 
on the basis of the reference frame data stored in the 
storage unit selected in the selection step and the 

25 frame data to be coded; a coding step of coding the 
frame data to be coded by using the motion vector 
estimated in the estimation step; and an output step of 
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outputting the coded data which is coded in the coding 
step. 

According to the present invention, the foregoing 
object is attained by providing a program for 
5 implementing control of a motion image coding apparatus 
which comprises a storage unit for storing a motion 
image, and codes the motion image by executing motion 
compensation for frame data to be coded by referring to 
frame data stored in the storage unit, comprising 

10 program codes of: an input step of inputting control 

information which controls an imaging device; a setting 
step of setting the number of reference frame data to 
be referred to when the frame data to be coded is 
coded, on the basis of motion information of the 

15 imaging device, which is acquired on the basis of the 
control information input in the input step; an 
acquisition step of acquiring reference frame data 
corresponding to the number of reference frame data set 
in the setting step; an estimation step of estimating a 

20 motion vector on the basis of the reference frame data 
acquired in the acquisition step and the frame data to 
be coded; a coding step of coding the frame data to be 
coded by using the motion vector estimated in the 
estimation step; and an output step of outputting the 

25 coded data which is coded in the coding step. 

Other features and advantages of the present 
invention will be apparent from the following 
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description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 

BRIEF DESCRIPTION OF DRAWINGS 
The accompanying drawings , which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
10 with the description, serve to explain the principles 
of the invention. 

Fig. 1 is a block diagram showing the arrangement 
of a motion image coding apparatus according to the 
first embodiment of the present invention; 
15 Fig. 2 is a flowchart showing a motion image 

coding process performed by the motion image coding 
apparatus according to the first embodiment of the 
present invention? 

Fig. 3 is a block diagram showing the arrangement 
20 of a motion image coding apparatus according to the 
second embodiment of the present invention; 

Fig. 4 is a flowchart showing a motion image 
coding process performed by the motion image coding 
apparatus according to the second embodiment of the 
25 present invention; 

Fig. 5 is a flowchart showing details of frame 
memory control M according to the second embodiment of 
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the present invention; 

Fig. 6 is a flowchart showing details of frame 
memory control N according to the second embodiment of 
the present invention; 
5 Fig. 7 is a block diagram showing the arrangement 

of a motion image coding apparatus according to the 
third embodiment of the present invention; 

Fig. 8 is a flowchart showing a motion image 
coding process performed by the motion image coding 
10 apparatus according to the third embodiment of the 
present invention; 

Fig. 9 is a block diagram showing the arrangement 
of a motion image coding apparatus according to the 
fourth embodiment of the present invention; 
15 Fig. 10 is a view showing the data configuration 

of a memory according to the fourth embodiment of the 
present invention; 

Fig. 11 is a flowchart showing processing 
executed by the motion image coding apparatus according 
20 to the fourth embodiment of the present invention; 

Fig. 12 is a flowchart showing details of 
processing in step S305 according to the fourth 
embodiment of the present invention; and 

Fig. 13 is a block diagram showing the 
25 arrangement of the conventional motion image coding 
apparatus . 
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BEST MODE FOR CARRYING OUT THE INVENTION 
Preferred embodiments of the present invention 
will be described in detail in accordance with the 
accompanying drawings. 
5 <First Embodiment > 

Fig, 1 is a block diagram showing the arrangement 
of a motion image coding apparatus according to the 
first embodiment of the present invention. 

The first embodiment will be explained below by 
10 taking a camcoder as an example of the motion image 
coding apparatus. 

In the first embodiment, a coding method used by 
the motion image coding apparatus will be explained by 
taking the H.264 coding method as an example. However, 
15 the coding method is not limited to this method. Also, 
to simplify the explanation, forward prediction by 
which frames in the past are referred to will be taken 
as an example. In addition, the maximum number of 
reference frames is set to 5 in order to simplify the 
20 explanation, but this number is not limited to 5. 

Referring to Fig. 1, an imaging device 1 
generates a digital signal of an image in cooperation 
with an optical unit made up of lenses and the like. A 
frame memory 2 stores the digital signal. A selector 3 
25 selects an output from the frame memory 2 in accordance 
with whether the coding mode is intra- frame coding or 
inter- frame coding. 
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An intra-predictor 4 executes intra-prediction by 
the H.264 coding method, A differential unit 5 
calculates a motion prediction error. A 
transformer/quantizer 6 executes integer type 
5 orthogonal transformation, and quantizes the obtained 
coefficient* A dequantizer/ invert transformer 10 
dequantizes the quantized coefficient, and executes 
integer type orthogonal inversion. 

An entropy coder 7 codes the quantization result 

10 from the transformer/quantizer 6. A recorder 8 records 
the coded data on a recording medium 9. The recording 
medium 9 records the coded data. An adder 11 adds the 
prediction error obtained by the dequantizer/invert 
transformer 10 and a predicted value (predicted image). 

15 Frame memories 12, 13, 14, 15, 16, and 17 store 

frames of locally decoded image data. A selector 18 
controls inputs and outputs. 

A motion estimator 19 extracts an optimum motion 
vector from the corresponding frame on the basis of the 

20 input image and decoded image. A motion compensator 20 
generates a predicted image from the motion vector 
calculated by the motion estimator 19 and the 
corresponding frame information. 

A motion coder 21 codes motion information on the 

25 basis of the calculated motion vector and the 

corresponding frame information. A motion detector 22 
detects the motion of the motion image coding apparatus 
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by using a gyro, sensor, or the like. A motion 
determinator 23 determines the speed of the motion, 
which is detected by the motion detector 22, of the 
motion image coding apparatus. 
5 The motion image coding apparatus shown in Fig. 1 

further comprises a CPU which controls the entire 
apparatus, a ROM which stores various control programs 
for controlling the apparatus, and a RAM which 
functions as a work area and temporary save area of 

10 various data for executing various control operations. 

The motion image coding operation of the motion 
image coding apparatus shown in Fig. 1 will be 
explained below. 

Before coding, the entropy coder 7 generates 

15 header information containing, e.g., the number of 
frames which can be referred to, and records this 
header information on the recording medium 9 via the 
recorder 8. 

Image data of frames sensed by the imaging device 
20 1 is stored in the frame memory 2, and input to the 

selector 3. The selecting operation of the selector 3 
is so controlled as to execute intra-frame coding at 
predetermined intervals, and execute inter-frame coding 
in other periods. 
25 First, the execution of intra-frame coding for 

the first frame as a start frame will be explained 
below. 
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The input frame data is input macroblock by 
macroblock to the intra-predictor 4, and the 
intra-predictor 4 executes intra -predict ion for each 
block. The prediction result is input to the 
5 transformer/quantizer 6, and the transformer /quantizer 
6 executes integer type orthogonal transformation, and 
quantizes the obtained coefficient. This quantized 
coefficient as the quantization result is input to the 
entropy coder 7 and dequantizer/ invert transformer 10. 

10 The entropy coder 7 entropy- codes the input 

quantization result, and records the coded data on the 
recording medium 9 via the recorder 8. 

The dequantizer/invert transformer 10 obtains a 
decoded image from the input quantization result, and 

15 inputs this decoded image to the adder 11. Intra-frame 
coding uses no predicted image. Therefore, the adder 
11 adds a predicted value 0, and stores the image in an 
empty frame memory or in a frame memory storing the 
oldest image data of reference images recorded in the 

20 frame memories. Initially, no image data is stored in 
the frame memories 12 to 17, so the image is stored in 
the frame memory 12. 

The execution of inter-frame coding for the 
subsequent second frame will be described below. 

25 The motion detector 22 detects the motion of the 

motion image coding apparatus itself. For example, the 
motion detector 22 calculates vertical and horizontal 
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motion vectors MVx and MVy, and defines the square sura 
of these vectors as the magnitude of the motion of the 
motion image coding apparatus. The motion determinator 
23 determines whether the magnitude of the motion of 
5 the motion image coding apparatus is larger than a 
predetermined value. 

If the magnitude of the motion of the motion 
image coding apparatus is larger than the predetermined 
value, the motion determinator 23 controls the selector 

10 18 so as to select outputs from the frame memories 12 
to 17 such that two temporally newest frames from the 
frame memories 12 to 17 are used as reference images. 
If the magnitude of the motion of the motion image 
coding apparatus is equal to or smaller than the 

15 predetermined value, the motion determinator 23 

controls the selector 18 so as to select all outputs 
from the frame memories 12 to 17. 

The motion estimator 19 reads out the contents of 
the selected frame memory, and calculates a motion 

20 vector. The motion compensator 20 generates a 
predicted image, and inputs this image to the 
differential unit 5. The selector 3 selects the 
differential unit 5 as an output. The differential 
unit 5 calculates a prediction error. The calculation 

25 result is inputted to the transformer/quantizer 6. 

Integer type orthogonal transf ormation is executed and 
. quantized the obtained coefficient at the 
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transformer/quantizer 6. The quantized coefficient as 
the quantization result is input to the entropy coder 7 
and dequantizer/ invert transformer 10. 

The entropy coder 7 entropy- codes the input 
5 quantization result , and records the coded data on the 
recording medium 9 via the recorder 8. Also, the 
motion vector calculated by the motion estimator 19 is 
coded by the motion coder 21 , and the coded data is 
recorded on the recording medium 9 via the recorder 8. 

10 The dequantizer /invert transformer 10 obtains a 

prediction error from the input quantization result, 
and inputs the prediction error to the adder 11. The 
adder 11 adds the predicted image (predicted value) 
from the motion compensator 20 to the prediction error, 

15 and stores the sum in an empty frame memory or in a 

frame memory storing the oldest image data of reference 
images recorded in the frame memories. 

The process flow of the motion image coding 
process of the motion image coding apparatus according 

20 to the first embodiment will be described below with 
reference to Fig. 2. 

Fig. 2 is a flowchart showing the motion image 
coding process of the motion image coding apparatus 
according to the first embodiment of the present 

25 invention. 

First, in step SI, header information is 
generated and output. In step S2, whether coding of 
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all frames is complete is determined. If coding is 
complete (YES in step S2), the process is terminated- 
If coding is not complete (NO in step S2), the flow 
advances to step S3. 
5 In step S3, frame data to be coded is input. In 

step S4, the motion of the motion image coding 
apparatus is detected. 

In step S5, the magnitude of the detected motion 
of the motion image coding apparatus is compared with a 

10 predetermined value. If the magnitude of the motion is 
equal to or smaller than the predetermined value (NO in 
step S5), the flow advances to step S6 to set the 
number of reference frames to M (in this case, 5). If 
the magnitude of the motion is larger than the 

15 predetermined value (YES in step S5), the flow advances 
to step S7 to set the number of reference frames to N 
(in this case, 2). In this embodiment, M > N always 
holds. 

In step S8, the coding mode of the frame data to 
20 be coded is determined. 

If the coding mode is intra-frarae coding, the 
flow advances to step S9 to execute intra-frame coding 
for the input frame data. After that, the flow 
advances to step Sll. 
25 If the coding mode is inter-frame coding, the 

flow advances to step S10 to execute inter-frame coding 
by executing motion compensation while images in frame 
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memories equal in number to the reference frames are 
referred to. After that, the flow advances to step 
Sll. 

In step Sll, the decoded image is stored in an 
5 empty frame memory or in a frame memory storing the 
temporally oldest image data of the frame memories. 

In step S12, the coded data is recorded on the 
recording medium 9. Then, the flow returns to step S2 
to process the next frame. 
10 In the first embodiment as described above, if 

the motion of the imaging device is large (lively), the 
frequency of reference of temporally separated frames 
extremely decreases, so these frames are not referred 
to. This largely reduces the arithmetic processing 
15 amount pertaining to motion compensation. As a 

consequence, the processing speed can be increased. 

In the first embodiment, the maximum number of 
reference frames is set to 5. However, it is obviously 
also possible to use any number of reference frames by 
20 using a necessary number of frame memories. 

Also, in the first embodiment, the number of 
reference frames when the motion is large is not 
limited to the above-mentioned number. In addition, 
the number of f rames to be selected may also be 
25 variable in accordance with the magnitude of the 

motion. For example, the number of frames which can be 
referred to can be determined inversely proportional to 
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the magnitude of the motion of the motion image coding 
apparatus, or some other standard may also be used. 

In the first embodiment, the motion of the 
imaging device 1 is detected by the motion detector 22 
5 (sensor). However, the motion of the imaging device 1 
may also be detected on the basis of input image data. 
For example, it is also possible to detect a rough 
motion vector by generating a reduced image, or detect 
the motions of several points of an image and use the 
10 detection results as the motion of the imaging device 
1. 

The first embodiment is explained by taking only 
forward prediction as an example. However, backward 
prediction or bidirectional prediction may also be 
15 performed by using a necessary number of frame 
memories . 

It is, of course, also possible to describe, in 
software, some or all of the various constituent 
elements of the motion image coding apparatus according 
20 to the first embodiment, and execute processing by an 
arithmetic unit such as a CPU. 
< Second Embodiment > 

Fig. 3 is a block diagram showing the arrangement 
of a motion image coding apparatus according to the 
25 second embodiment of the present invention. 

The same reference numerals as in Fig. 1 of the 
first embodiment denote parts having the same functions 
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in Fig. 3, and an explanation thereof will be omitted. 

Referring to Fig. 3, a motion estimator 100 
refers to an input image and frame data which can be 
referred to from decoded image data in frame memories 
5 12 to 17, and extracts an optimum motion vector from 
the corresponding frame data. 

A motion compensator 101 generates a predicted 
image from the motion vector calculated by the motion 
estimator 100 and the corresponding frame information. 
10 A frame memory controller 102 controls the frame 
memories 12 to 17. That is, the frame memory 
controller 102 controls write and read and the supply 
of power to the frame memories 12 to 17. 

The motion image coding operation of the motion 
15 image coding apparatus shown in Fig. 3 will be 
described below. 

First, as in the first embodiment, header 
information is generated and recorded on a recording 
medium 9 via a recorder 8. 
20 Then, as in the first embodiment, image data of 

frames sensed by an imaging device 1 are stored in a 
frame memory 2 and input to a selector 3. 

When intra -frame coding is to be executed, 
processing is entirely the same as in the first 
25 embodiment, and the generated coded data is recorded on 
the recording media 9 via the recorder 8. 

In addition, in accordance with instructions from 
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the frame memory controller 102, a decoded image is 
stored in an empty frame memory or in a frame memory 
storing the oldest image data of reference images 
recorded in the frame memories. That is, the frame 
5 memory controller 102 supplies power to one of unused 
frame memories to set it in a writable state , or sets a 
frame memory having the oldest frame of the frame 
memories in a writable state. 

The execution of inter- frame coding will be 

10 explained below. 

As in the first embodiment, a motion detector 22 
detects the motion of the motion image coding apparatus 
itself. For example, the motion detector 22 calculates 
vertical and horizontal motion vectors MVx and MVy, and 

15 defines the square sum of these vectors as the 

magnitude of the motion of the motion image coding 
apparatus. A motion determinator 23 determines whether 
the magnitude of the motion of the motion image coding 
apparatus is larger than a predetermined value. 

20 If the magnitude of the motion of the motion 

image coding apparatus is larger than the predetermined 
value, the motion determinator 23 notifies the frame 
memory controller 102 that the motion image coding 
apparatus has moved largely. When receiving this 

25 notification, the frame memory controller 102 sets two 
of the frame memories 12 to 17 in a readable state, and 
sets another frame memory in a writable state, in order 
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to selects outputs from the frame memories 12 to 17 so 
that the two temporally newest frames are chosen as 
reference images. In addition, the frame memory 
controller 102 stops the supply of power to the other 
5 frame memories. 

If the magnitude of the motion of the motion 
image coding apparatus is equal to or smaller than the 
predetermined value, the motion determinator 23 
notifies the frame memory controller 102 that the 

10 motion image coding apparatus has not largely moved. 
When receiving this notification, the frame memory 
controller 102 sets a frame memory storing the frame 
data in a readable state. Also, the frame memory 
controller 102 supplies power to one of unused frame 

15 memories to set it in a writable state, or sets a frame 
memory having the oldest frame of the frame memories in 
a writable state. 

The motion estimator 100 reads out the contents 
of the fjrame memory set in the readable state, and 

20 calculates a motion vector. The motion compensator 101 
generates a predicted image, and inputs the image to a 
differential unit 5. After that, as in the first 
embodiment, inter -frame coding is performed, and the 
coded data is recorded on the recording medium 9 via 

25 the recorder 8. 

A deguantizer/ invert transformer 10 obtains a 
prediction error from the input quantization result. 
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and inputs this prediction error to an adder 11. The 
adder 11 adds the predicted image from the motion 
compensator 101 to the prediction error, and stores the 
addition result in the frame memory set in the writable 
5 state . 

The process flow of the motion image coding 
process of the motion image coding apparatus according 
to the second embodiment will be described below with 
reference to Figs. 4 to 6. 
10 Fig. 4 is a flowchart showing the motion image 

coding process of the motion image coding apparatus 
according to the second embodiment of the present 
invention. 

The same reference numerals as in Fig . 2 of the 
15 first embodiment denote steps having the same functions 
in Fig. 4, and an explanation thereof will be omitted. 

As in the first embodiment, after processes in 
steps SI to S4, if the magnitude of the motion is equal 
to or smaller than the predetermined value in step S5 
20 (NO in step S5), the flow advances to step S100 to 
execute frame memory control M. If the magnitude of 
the motion is larger than the predetermined value (YES 
in step S5), the flow advances to step S101 to execute 
frame memory control N. In this embodiment, M > N 
25 always holds. 

Details of frame memory control M in step S100 
will be explained below with reference to Fig. 5. 
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Fig. 5 is a flowchart showing the details of 
frame memory control M according to the second 
embodiment of the present invention. 

In step S102 r the number of reference frames is 
5 set to M (in this case, 5). In step S103, the 

presence/absence of unused frame memories which are 
supplied with no power and kept unused is determined. 
If there is no unused frame memory (NO in step S103), 
the flow advances to step S104 to set a frame memory 
10 storing the temporally oldest image data (reference 
frame) in a writable state. 

On the other hand, if unused frame memories exist 
(YES in step S103), the flow advances to step S105 to 
select one unused frame memory and make it usable by 
15 supplying power to it. In step S106, this frame memory 
made usable in step S105 is set in a writable state. 

In step S107, a maximum of M frame memories from 
the temporally newest one of the remaining frame 
memories are set in a readable state. In this manner, 
20 the sequence of frame memory control M is complete, and 
the flow advances to step S8 in Fig. 4. 

Details of frame memory control N in step S101 
will be explained below with reference to Fig. 6. 

Fig. 6 is a flowchart showing the details of 
25 frame memory control N according to the second 
embodiment of the present invention. 

In step S110, the number of reference frames is 
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set to N (in this case, 2). 

In step Sill, N frame memories from the 
temporally newest one of the remaining frame memories 
are set in a readable state. In step S112, one of the 
5 remaining frame memories is set in a writable state. 

In step S113, power supply to the still remaining 
frame memories is stopped because they are unused frame 
memories. In this way, the sequence of memory control 
N is complete, and the flow advances to step S8. 
10 Referring back to Fig. 4, as in the first 

embodiment, after processes in steps S8 to S10, in step 
S102 the decoded image is stored in a writable one of 
the frame memories. In step S12, the coded data is 
recorded on the recording medium 9, and the flow 
15 returns to step S2 to process the next frame. 

In the second embodiment as described above, in 
addition to the effects explained in the first 
embodiment, the power consumption can be reduced by 
stopping power supply to unused frame memories. 
20 Therefore, especially in an environment in which the 
apparatus is operated by a limited power supply, the 
operation time of the apparatus can be prolonged. 

In the second embodiment, the maximum number of 
reference frames is set to 5. However, it is obviously 
25 also possible to use any number of reference frames by 
using a necessary number of frame memories. 

Also, in the second embodiment, the number of 
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reference frames when the motion is large is not 
limited to the above-mentioned number. In addition, 
the number of frames to be selected may also be 
variable in accordance with the magnitude of the 
5 motion. For example, the number of frames which can be 
referred to can be determined Inversely proportional to 
the magnitude of the motion of the motion image coding 
apparatus, or some other standard may also be used. 

The second embodiment is explained by taking only 
10 forward prediction as an example. However, backward 
prediction or bidirectional prediction may also be 
performed by using a necessary number of frame 
memories. 

It is, of course, also possible to describe, in 
15 software, some or all of the various constituent 

elements of the motion image coding apparatus according 

to the second embodiment, and execute processing by an 

arithmetic unit such as a CPU. 

<Third Embodiment > 
20 Fig. 7 is a block diagram showing the arrangement 

of a motion image coding apparatus according to the 

third embodiment of the present invention. 

The same reference numerals as in Fig. 3 of the 

second embodiment denote parts having the same 
25 functions in Fig. 7, and an explanation thereof will be 

omitted. 

Referring to Fig. 7, an image sensing mode 
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setting unit 200 sets the image sensing mode of the 
motion image coding apparatus. Examples of the image 
sensing mode are "automatic mode" as a standard, 
"sports mode" for sensing an image of an object having 
5 a high speed, and "scenery mode" for sensing images of 
scenes such as mountains, seas, and the like. However, 
the image sensing modes are not lim±ted to these 
examples . 

A frame memory controller 201 controls frame 

10 memories 12 to 17. In accordance with the image 

sensing mode, the frame memory controller 201 controls 
write and read and power supply to the frame memories 
12 to 17. An imaging device 202 can control the 
shutter speed, aperture value, and the like in 

15 accordance with instructions from the image sensing 
mode setting unit 200. 

The motion image coding operation of the motion 
image coding apparatus shown in Fig. 7 will be 
described below ♦ 

20 First, a user (not shown) sets the image sensing 

mode of the imaging device 202 by using the image 
sensing mode setting unit 200 before starting image 
sensing. The set image sensing mode is input to the 
imaging device 202 and frame memory controller 201. 

25 The imaging device 202 selects and sets a shutter speed 
and F-number suited to the set image sensing mode. 
When ^automatic mode" is selected, the frame 
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memory controller 201 sets the number of reference 
frames to 3, and stops power supply to the frame 
memories 16 and 17 to keep them unused. When "sports 
mode" is selected, the frame memory controller 201 sets 
5 the number of reference frames to 1, and stops power 
supply to the frame memories 14 to 17 to keep them 
unused. When "scenery mode" is selected, the frame 
memory controller 201 sets the number of reference 
frames to 5. 

10 After that, intra-fraroe coding or inter-frame 

coding of each frame is executed by using frame 
memories to which electric power is supplied. 

When intra- frame coding is to be executed, the 
frame memory controller 201 sets, from the frame 

15 memories to which power is supplied, an empty frame 

memory or a frame memory storing the temporally oldest 
frame in a writable state in order, and stores the 
decoded image. 

When inter- frame coding is to be executed, the 

20 frame memory controller 201 sets, from of the frame 
memories to which power is supplied, an empty frame 
memory or a frame memory storing the temporally oldest 
frame in a writable state in order, and sets the other 
frame memories in a readable state - 

25 Note that in inter-frame coding, the number of 

reference frames is set for each image sensing mode. 
Therefore, in accordance with the number of reference 
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frames, motion vectors are calculated, and motion 
compensation is executed. The generated coded data is 
recorded on a recording medium 9 via a recorder 8. The 
decoded image is stored in the frame memories set in 
5 the writable state. 

The process flow of the motion image coding 
process of the motion image coding apparatus according 
to the third embodiment will be described below with 
reference to Pig. 8. 
10 Tig. 8 is a flowchart showing the motion image 

coding process of the motion image coding apparatus 
according to the third embodiment of the present 
invention . 

The same reference numerals as in Fig. 2 of the 
15 first embodiment denote steps having the same functions 
in Fig. 8, and an explanation thereof will be omitted. 
In step S200, the image sensing mode is set. 
Assume that in Fig. 8, "sports mode", "scenery 
mode", and "automatic mode" are the image sensing 
20 modes . 

In step S201, whether the image sensing mode is 
"sports mode" is determined. If the image sensing mode 
is "sports mode" (YES in step S201), the flow advances 
to step S202 to set the number of reference frames to N 
25 (in this case, 1). in step S203, power supply to 
unused frame memories are stopped. After that, the 
flow advances to step SI . 
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If the image sensing mode is not " sports mode" in 
step S201 (NO in step S201), the flow advances to step 
S204 to determine whether the image sensing mode is 
"scenery mode". If the image sensing mode is "scenery 
5 mode", the flow advances to step S205 to set the number 
of reference frames to P (in this case, 5). After 
that, the flow advances to step SI. 

If the image sensing mode is not "scenery mode" 
(NO in step S204), the flow advances to step S206 to 
10 set the number of reference frames to M (in this case, 
3). In step S207, power supply to unused frame 
memories is stopped. The flow then advances to step 
SI. In this embodiment, P s= M a N always holds. 

After that, as in the first embodiment, processes 
15 in steps SI to S3 and S8 to S12 are executed on the 

basis of one of the set numbers of reference frames P, 
M, and N. 

In the third embodiment as described above, the 
number of reference frames is controlled on the basis 

20 of the magnitude, which is expected by the image 

sensing mode, of the motion of a motion image to be 
coded. As a consequence, optimum processing can be 
executed in accordance with the motion. 

Also, since the number of reference frames is 

25 preset on the basis of the image sensing mode, optimum 
reference can be performed. In addition, the power 
consumption can be reduced by stopping power supply to 
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unused frame memories. Therefore, especially in an 
environment in which the apparatus is operated by a 
limited power supply, the operation time of the 
apparatus can be prolonged. 
5 In the third embodiment, the maximum number of 

reference frames is set to 5. However, it is obviously 
also possible to use any number of reference frames by 
using a necessary number of frame memories. 

Also, the third embodiment is explained by taking 
10 only forward prediction as an example. However, 

backward prediction or bidirectional prediction may 
also be performed by using a necessary number of frame 
memories . 

It is, of course, also possible to describe, in 
15 software, some or all of the various constituent 

elements of the motion image coding apparatus according 
to the third embodiment, and execute processing by an 
arithmetic unit such as a CPU. 
<Fourth Embodiment > 
20 Fig. 9 is a block diagram showing the arrangement 

of a motion image coding apparatus according to the 
fourth embodiment of the present invention. 

Referring to Fig. 9, a central processing unit 
(CPU) 300 controls the entire apparatus and performs 
25 various processes. A memory 301 provides storage areas 
for an operating system (OS), software, and arithmetic 
operations necessary to control this apparatus. A bus 



WO 2005/062624 



PCT/JP2004/018666 



- 37 - 

302 interconnects various constituent elements forming 
the motion image coding apparatus, and transfers data 
and control signals. 

A terminal 303 is used to activate the apparatus, 

5 set various conditions, and execute playback 

instructions. A storage device 304 stores software. A 
storage deviqe 305 stores coded streams. The storage 
devices 304 and 305 may also be portable storage media 
which can be separated from the motion image coding 

0 apparatus and moved. 

A camera (imaging device) 307 can sense motion 
images frame by frame. A camera panhead 306 of the 
camera 307 is controlled by software, and has a 
function of outputting, to the bus 302, image data from 

5 the camera 307, the state of the camera 307, and the 
state of the camera panhead 306 itself. 

A monitor 308 displays images. A communication 
line 310 is, e.g., a LAN, public line, radio channel, 
or broadcasting radio wave. A communication interface 

0 (I/F) 309 transmits and receives streams via the 
communication line 310. 

The memory 301 stores the OS for controlling the 
entire apparatus and operating various software, and 
also stores the software to be operated. In addition, 

5 the memory 301 has an image area for storing image 

data, a code area for storing generated coded data, and 
a working area for storing, e.g., parameters for 
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various arithmetic operations and coding. 

A process which, in the arrangement as described 
above, controls the camera panhead 306 and the camera 
307 from an external terminal via the communication 
5 line 310, codes a motion image sensed by the camera 
307, and transmits the coded motion image to the 
external terminal via the communication line 310, will 
be described below. 

In the fourth embodiment, a coding method used by 

10 the motion image coding apparatus will be explained by 
taking the H.264 coding method as an example. However, 
any coding method can be used as long as an image is 
coded by referring to two or more frames. Also, to 
simplify the explanation, three frames before a frame 

15 to be coded and three frames after it are referred to. 
However, the number of reference frames is not limited. 

Before the processing, the terminal 303 instructs 
the whole apparatus to perform activation, thereby 
initializing the individual constituent elements. 

20 Consequently, software stored in the storage device 304 
is developed on the memory 301 via the bus 302, and 
activated. 

The data configuration of the memory 301 will be 
explained below with reference to Fig. 10. 
25 Fig. 10 is a view showing the data configuration 

of the memory according to the fourth embodiment of the 
present invention. 
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As shown in Fig. 10, the memory 301 stores the OS 
for controlling the whole apparatus and operating 
various software , motion image coding software for 
executing the H.264 coding method, communication 
5 software for controlling communication, and camera 
control software for controlling the camera panhead 
306. The memory 301 also has image areas 1 to 7 for 
storing image data, a code area for storing coded data, 
and a working area. 
10 The way the CPU 300 controls the motion image 

coding apparatus in the arrangement as described above 
will be explained below with reference to Pig. 11. 

Fig. 11 is a flowchart showing processing 
executed by the motion image coding apparatus according 
15 to the fourth embodiment of the present invention. 

In step S300, the CPU 300 activates the camera 
control software to initialize the individual parts, 
and starts sensing motion images. In step S301, the 
CPU 300 activates the communication software to receive 



20 camera control~^nf ormation via the communication— line — 

310, thereby making preparations for outputting coded 
data. 

In step S302, the CPU 300 determines whether 
process termination is input from the terminal 303. If 
25 termination is input (YES in step S302), the flow 

advances to step S307, and the CPU 300 terminates the 
communication software. In step S308, the CPU 300 
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terminates the camera control software r and then 
terminates the entire process. 

On the other hand, if termination is not input in 
step S302 (NO in step S302), the flow advances to step 
5 S303, and the CPU 300 determines whether a camera 

control request and motion image transmission request 
are input from the communication line 310 via the 
communication interface 309. If no such requests are 
input (NO in step S303), the flow returns to step S302, 
10 and the CPU 300 waits until these requests are input. 
If these requests are input (YES in step S303), the 
flow advances to step S304. 

In step S304, the CPU 300 activates the motion 
image coding software. In step S305, the CPU 300 
15 executes coding and transmits the coded data. Details 
of this process will be described later. When the 
communication is complete, the flow advances to step 
S306. In step S306, the CPU 300 terminates the coding 
software. The flow then returns to step S302, and the 
20 CPU 300 waits for the next instruction. 

The details of the process in step S305 will be 
explained below with reference to Fig. 12. 

Fig. 12 is a flowchart showing the details of the 
process in step S305 according to the fourth embodiment 
25 of the present invention. 

First, in step S351, header information is 
generated and stored in the code area on the memory 
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301 . When the coded data is stored in the code area, 
the communication software transmits the data to the 
communication line 310 via the communication interface 
309, and, after the transmission, clears the 
5 corresponding region of the code area. Transmission of 
the coded data in the code area will not particularly 
be described in the following explanation. 

In step S352, the CPU 300 determines the 
presence/absence of a communication termination 
10 request. If there is a communication termination 

request (YES in step S352), the CPU 300 terminates the 
process. If there is no communication termination 
request (NO in step S352), the flow advances to step 
S353. 

15 In step S353, if a camera control command for 

controlling the camera is received from the 
communication line 310 via the communication interface 
309, the CPU 300 inputs the command to the camera 
panhead 306. On the basis of this camera control 

20 command, the camera panhead 306 performs operations 
such as panning, tilting, and zooming. 

In step S354, the CPU 300 inputs frame data taken 
by the camera 307 via the camera panhead 306, and 
stores the data in image area 1 on the memory 301. 

25 In step S355, the CPU 300 acquires data 

concerning the motion of the camera 307 from the camera 
panhead 306. That is, the camera panhead 306 operates 
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the camera 307 after receiving the camera control 
command, and data concerning the motion of the camera 
307 can be acquired by monitoring the status of a motor 
or the like. The CPU 300 writes the acquired data in 
5 the working area on the memory 301. 

In step S356, the CPU 300 compares the obtained 
data (the magnitude of the motion) pertaining to the 
motion of the camera 307 with a predetermined value. 
If the magnitude of the motion is equal to or 

10 smaller than the predetermined value (NO in step S356), 
the flow advances to step S357, and the CPU 300 sets 
the number of front and back reference frames to M (in 
this case, 3). Then, in step S358, the CPU 300 sets m 
x m as the range of search for motion vectors to be 

15 used when motion compensation is performed during 

inter-frame coding (to be described later). In this 
case, m is, e.g., 32 pixels. After that, the flow 
advances to step S361. 

If the magnitude of the motion is larger than the 

20 predetermined value (YES in step S356), the flow 

advances to step S359, and the CPU 300 sets the number 
of front and back reference frames to N (in this case, 
1). In this embodiment, M > N always holds. In step 
S360, the CPU 300 sets n x n as the range of search for 

25 motion vectors to be used when motion compensation is 
performed during inter-frame coding (to be described 
later). In this case, n is, e.g., 55 pixels. After 
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that, the flow advances to step S361. 

In step S361, the CPU 300 determines the coding 
mode of frame data to be coded. 

If the coding mode is intra-frame coding , the 
5 flow advances to step S362, and the CPU 300 executes 
intra-frame coding for the input frame data. The CPU 
300 then stores the generated coded data in the code 
area on the memory 301 , and the flow advances to step 
S364. 

10 If the coding mode is inter-frame coding, the 

flow advances to step S363, and the CPU 300 executes 
inter-frame coding by executing motion compensation by 
searching for motion vectors within the set range, 
while referring to an image in an image area having the 

15 set number of reference frames. The CPU 300 then 

stores the generated coded data in the code area on the 
memory 301 f and the flow advances to step S364. 

In step S364 r the CPU 300 stores the decoded 
image in an empty image area or in an image area 

20 storing the temporally oldest image data of the image 
areas on the memory 301. In step S365 f the CPU 300 
transmits the coded data stored in the code area to the 
communication line 310 via the communication interface 
309, and the flow returns to step S352 to process the 

25 next frame. 

In the fourth embodiment as described above, if 
the motion of the camera is large (lively), temporally 



WO 2005/062624 



PCT/JP2004/018666 



- 44 - 

separated frames are not referred to because the 
reference frequency of these frames extremely 
decreases. In this manner, the arithmetic processing 
amount of motion compensation can be greatly reduced. 
5 As a consequence, the processing speed can be 
increased. 

Also, if the motion of the camera is large 
(lively) , accurate motion vector search can be 
performed by making the motion vector search range 

10 wider than that when the camera motion is small. That 
is, when the number of reference frames is reduced from 
5 to 2, the arithmetic operation amount remains the 
same even if the motion vector search range is extended 
to (3025(55 x 55))/1024(32 x 32)) 1/2 . This makes it 

15 possible to efficiently search for motion vectors 
without increasing the load on the CPU. 

In the fourth embodiment, reference frames are 
three frames before a frame to be coded and three 
frames after it. However, it is obviously also 

20 possible to use any number of reference frames by 

securing a necessary amount of image areas for storing 
reference frames . 

In the fourth embodiment, data is transmitted to 
the communication line 310 via the communication 

25 interface 309 by activating the communication software. 
However, data may also be stored in the storage device 
305. 
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In addition , input images need not be sensed 
images from the camera. It is also possible to use 
motion image data stored in the storage device 305, and 
change the number of reference frames while determining 
5 the magnitude of the motion of the frame. Furthermore, 
the number of reference frames may also be changed if a 
scene change or the like is detected* 

It is, of course, also possible to implement 
embodiments by freely combining the arrangements of the 
10 first to fourth embodiments in accordance with the 
applications or purposes. 

As many apparently widely different embodiments of 
the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
15 understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
appended claims. 

CLAIM OF PRIORITY 
20 This application claims priority from Japanese 

Patent Application No. 2003-425826 filed on December 
22, 2003, which is hereby incorporated by reference 
herein . 



25 



